Imaging device, image processing system, and image processing method

ABSTRACT

An imaging device includes a plurality of image sensors configured to capture a plurality of images, and processing circuitry. The processing circuitry is configured to determine whether a difference in brightness between the plurality of images exceeds a predetermined threshold. Based on a determination that the difference in brightness exceeds the predetermined threshold, the processing circuitry arranges the plurality of images side by side in one direction to be output as an output image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application No. 2019-119671, filed inJapan on Jun. 27, 2019, in the Japan Patent Office, the entiredisclosure of which is hereby incorporated by reference herein.

BACKGROUND Technical Field

Embodiments of the present application described herein provide animaging device, an imaging processing system, and an image processingmethod.

Background Art

An omnidirectional imaging system that uses a plurality of wide-anglelenses such as fish-eye lenses and super-wide-angle lenses to create anomnidirectional image from a plurality of images is known. Hereinafter,such an omnidirectional image is referred to as a spherical image.However, since the plurality of images are taken by a plurality oflenses, the omnidirectional image looks unnatural depending on thesubject taken by each lens and the surrounding conditions.

SUMMARY

In one aspect of this disclosure, there is provided an improved imagingdevice including a plurality of image sensors configured to capture aplurality of images, and processing circuitry. The processing circuitryis configured to determine whether a difference in brightness betweenthe plurality of images exceeds a predetermined threshold. Based on adetermination that the difference in brightness exceeds thepredetermined threshold, the processing circuitry arranges the pluralityof images side by side in one direction to be output as an output image.

In another aspect of this disclosure, there is provided an improvedimage processing system including processing circuitry configured to:acquire a plurality of images captured by a plurality of image sensors;determine whether a difference in brightness between the plurality ofimages exceeds a predetermined threshold; based on a determination thatthe difference in brightness exceeds the predetermined threshold,arrange the plurality of images side by side in one direction togenerate an output image.

In still another aspect of this disclosure, there is provided animproved image processing method including: acquiring a plurality ofimages and determining whether a difference in brightness between theplurality of images exceeds a predetermined threshold. In a case thatthe determining determines that the difference in brightness exceeds thepredetermined threshold, the method further includes arranging theplurality of images side by side in one direction to generate an outputimage.

BRIEF DESCRIPTION OF THE DRAWINGS

The aforementioned and other aspects, features, and advantages of thepresent disclosure will be better understood by reference to thefollowing detailed description when considered in connection with theaccompanying drawings, wherein:

FIG. 1 is a diagram of an omnidirectional imaging system 1.

FIG. 2 is a sectional view of an omnidirectional imaging device 10.

FIG. 3 is a block diagram of a hardware configuration of theomnidirectional imaging device 10.

FIG. 4 is a block diagram of a hardware configuration of informationprocessing device 50 of the omnidirectional imaging system 1.

FIG. 5 is a flow chart of an image processing of the omnidirectionalimaging device 10.

FIG. 6A is a diagram of a data structure and data flow in a generationmethod for generating an omnidirectional image.

FIG. 6B illustrates a plane data structure of image data of anomnidirectional image.

FIG. 6C illustrates a spherical data structure of image data of anomnidirectional image.

FIG. 7A is a diagram illustrating a definition of attitude viewed from aside of the omnidirectional imaging device 10.

FIG. 7B is a diagram illustrating a definition of attitude viewed from afront of the omnidirectional imaging device 10.

FIG. 8A is a diagram of a spherical data structure before a zenith androtation correction.

FIG. 8B is the spherical data structure after the zenith and rotationcorrection.

FIG. 8C is an omnidirectional image before the zenith and rotationcorrection.

FIG. 8D is the omnidirectional image after the zenith and rotationcorrection.

FIG. 9A is a diagram illustrating an arrangement of partial images of anomnidirectional image.

FIG. 9B is an omnidirectional image before application of a high dynamicrange (“HDR”) processing.

FIG. 9C is the omnidirectional image after application of the HDRprocessing.

FIG. 10 is a diagram illustrating a functional configuration of theomnidirectional imaging device 10.

FIG. 11 illustrates a process performed by the omnidirectional imagingdevice 10.

FIG. 12A is a diagram illustrating an arrangement of partial images ofan omnidirectional image.

FIGS. 12B and 12C are exemplary omnidirectional images captured in abrightness difference scene mode.

FIGS. 13A-13C illustrate orientations of an omnidirectional imagingdevice with respect to zenith correction of orientations of objects incaptured omnidirectional images.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this patent specification is not intended to be limited to thespecific terminology so selected and it is to be understood that eachspecific element includes all technical equivalents that have the samefunction, operate in a similar manner, and achieve similar results.

Although the embodiments are described with technical limitations withreference to the attached drawings, such description is not intended tolimit the scope of the disclosure and all of the components or elementsdescribed in the embodiments of this disclosure are not necessarilyindispensable.

The present disclosure is not limited to the following embodiments, andthe constituent elements of the embodiments includes those which can beeasily conceived by those skilled in the art, substantially the sameones, and those in the following embodiments include those which can beeasily conceived by those skilled in the art, substantially the same,and within equivalent ranges. Furthermore, various omissions,substitutions, changes and combinations of constituent elements can bemade without departing from the gist of the following embodiments.

Exemplary implementations of the present application are describedbelow, but no limitation is indicated therein, and various applicationsand modifications may be made without departing from the scope of theapplication. In the implementations described below, as illustrated inFIG. 1, an omnidirectional imaging system 1 includes an omnidirectionalimaging device 10 including two fisheye lenses and an informationprocessing device 50 communicating with the omnidirectional imagingdevice 10.

Hereinafter, the schematic configuration of the omnidirectional imagingsystem 1 according to the present implementation is described withreference to FIG. 1 and FIG. 4. FIG. 1 is a diagram of theomnidirectional imaging system 1. The omnidirectional imaging system 1includes the omnidirectional imaging device 10 and the informationprocessing device 50. The omnidirectional imaging device 10 captures aplurality of images, such as wide-angle lens images, fish-eye lensimages, and the like. The omnidirectional imaging device 10 is a devicethat captures images to generate an omnidirectional image centered onthe omnidirectional imaging device 10.

The information processing device 50 is a terminal that communicateswith the omnidirectional imaging device 10 and performs operations onthe omnidirectional imaging device 10, reception of captured images, andthe like. In FIG. 1, a smartphone terminal is shown as an example of theinformation processing device 50, but information processing device 50is not particularly limited. Details of the omnidirectional imagingdevice 10 and the information processing device 50 will be describedlater.

FIG. 2 is a sectional view of the omnidirectional imaging device 10. Theomnidirectional imaging device 10 illustrated in FIG. 2 includes animaging body 12 and a housing 14 that holds the imaging body 12. Theomnidirectional imaging device 10 may further include other componentssuch as circuitry or processing circuitry (to be described below), acontroller board, a battery, and a shutter button 18 that is provided onthe housing 14.

The imaging body 12 illustrated in FIG. 2 includes two lens systems 20Aand 20B and two solid-state image sensors 22A and 22B. Each of thesolid-state image sensors 22A and 22B may be, for example, acharge-coupled device (CCD) sensor or a complementary metal oxidesemiconductor (CMOS) sensor. The solid-state image sensors 22A and 22Bare set so that the imaging surfaces are opposite to each other. Forexample, the two solid-state image sensors 22A and 22B are provided as aplurality of imaging units for the two lens systems 20A and 20B.However, the omnidirectional imaging device 10 of the presentapplication is not limited to these. In other implementations, eachdifferent part of one solid-state image sensor may be used as an imagingunit, and an image may be formed on each part of one image sensorthrough the plurality of lens systems 20A and 20B. Each of the lenssystems 20A and 20B may be configured as a fish-eye lens consisting of,for example, seven elements in six groups, fourteen elements in tengroups. In the omnidirectional imaging device 10 illustrated in FIG. 2,the above-mentioned fish-eye lens has a full angle of view of largerthan 180 degrees (=360 degrees/n, where n denotes the number of opticalsystems and n is 2), and preferably has an angle of view of 190 degreesor larger. In the implementation to be described, two fisheye lenseshaving a total angle of view of more 180 degrees are used. However, itis not limited to these, but as long as a predetermined angle of view isobtained as a whole, three or more lens optical systems and imagesensors may be used. Moreover, the omnidirectional imaging device 10 isnot limited to fisheye lenses, and may include other types of lensessuch as wide-angle lens or super-wide-angle lenses as long as apredetermined angle of view is obtained as a whole.

The relative positions of the optical elements (lenses, prisms, filters,and aperture stops) of lens system 20A and lens system 20B aredetermined with reference to the corresponding solid-state image sensor22A and solid-state image sensor 22B. More specifically, positioning ismade such that the optical axis of the optical elements of each of thelens system 20A and 20B is positioned at the central part of the lightreceiving area of a corresponding one of the solid-state image sensor22A and 22B orthogonally to the light receiving area, and such that thelight receiving area serves as the imaging plane of corresponding one ofthe fish-eye lenses. In order to reduce the parallax, folded optics maybe adopted. Folded optics is a system in which light, converged by twolens systems 20A and 20B, can be divided to two image sensors by the tworectangular prisms. However, the present application is not limited tothis configuration, and a three-fold refraction structure may be used inorder to further reduce parallax, or a straight optical system may beused to reduce costs.

In the implementation illustrated in FIG. 2, the lens systems 20A and20B have the same specifications and are oriented in directions reverseto each other such that the optical axes thereof coincide with eachother. The solid-state image sensors 22A and 22B convert the lightdistribution of the received light into an image signal, andsequentially output image frames to the image processing block of thecontroller board. As will be described later in detail, the imagescaptured by the respective solid-state image sensor 22A and 22B arecombined so as to generate an image over a solid angle of 4π steradian(hereinafter, such an image is referred to as a “spherical image” or“omnidirectional image”). The spherical image is obtained byphotographing multiple images, all the directions viewable from aphotographing location and combining the photographed images. While itis assumed in the example implementation described below that aspherical image is to be generated, a so-called panoramic image obtainedby photographing 360 degrees only in a horizontal plane or an image thatis a part of the image obtained by photographing omnidirectionally or360 degrees in a horizontal plane may also be generated. (For example, afull sky (dome) image taken 360 degrees horizontally and 90 degreesvertically from the horizon) The spherical image may be obtained as astill image or as moving images.

FIG. 3 is a block diagram of the hardware configuration of theomnidirectional imaging device 10. The omnidirectional imaging device 10comprises a digital still camera processor 100 (hereinafter, simplyprocessor), a lens barrel unit 102, and various elements connected withthe processor 100. The lens barrel unit 102 includes the two pairs oflens systems 20A, 20B and solid-state image sensors 22A, 22B. Thesolid-state image sensors 22A, 22B are controlled by a command from acentral processing unit (CPU) 130 of the processor 100. The CPU 130 willbe described later in detail. Moreover, processor 100 and CPU 130,either separately or together, may be referred to as circuitryprocessing circuitry.

The circuitry or processing circuitry may include general purposeprocessors, special purpose processors, integrated circuits, ASICs(“Application Specific Integrated Circuits”), conventional circuitry,CPUs, controllers, and/or combinations thereof which are configured orprogrammed to perform the disclosed functionality. Processors andcontrollers are considered processing circuitry or circuitry as theyinclude transistors and other circuitry therein. In this disclosure, anycircuitry, units, controllers, or means are hardware carry out or areprogrammed to perform the recited functionality. The hardware may be anyhardware disclosed herein or otherwise known which is programmed orconfigured to carry out the recited functionality. When the hardware isa processor or controller which may be considered a type of circuitry,the circuitry, means, or units are hardware and/or the hardware andprocessor may be configured by executable instructions as described inthis application.

The processor 100 includes Image Signal Processors (ISP) 108, DirectMemory Access Controllers (DMAC) 110, and an arbiter (ARBMEMC) 112 forarbitrating a memory access. In addition, the processor 100 includes aMemory Controller (MEMC) 114 for controlling the memory access, adistortion correction and image synthesis block 118, and a shooting modeswitching unit 201. The ISPs 108A and 108B respectively performsAutomatic Exposure (AE) control, Automatic white balance (AWB) setting,and gamma setting on images input through signal processing by thesolid-state image sensors 22A and 22B. In FIG. 3, two ISPs 108A and 108Bare provided corresponding to the two solid-state image sensors 22A and22B. However, it is not limited to these, but one ISP may be providedfor the two solid-state image sensors 22A and 22B.

The MEMC 114 is connected to an SDRAM 116 which temporarily stores dataused in the processing of the ISP 108A, 108B and distortion correctionand image synthesis block 118. The distortion correction and imagesynthesis block 118 performs distortion correction and verticalcorrection on the two partial images from the two pairs of the lenssystems 20 and the solid-state image sensor 22 on the basis ofinformation from a motion sensor 120 and synthesizes them. The motionsensor 120 may include a triaxial acceleration sensor, a triaxialangular velocity sensor, a geomagnetic sensor, and the like. A facedetection block 119 performs face detection from the image and specifiesthe position of the person's face. In addition to the face detectionblock 119 or instead of the face detection block 119, an objectrecognition block for recognizing other subjects such as a full bodyimage of a person, a face of an animal such as a cat or dog, a car or aflower may be provided.

The processor 100 further comprises a DMAC 122, an image processingblock 124, a CPU 130, an image data transferrer 126, an SDRAMC 128, amemory card control block 140, a USB block 146, a peripheral block 150,an audio unit 152, a serial block 158, an LCD (Liquid Crystal Display)driver 162, and a bridge 168.

The CPU 130 controls the operations of the elements of theomnidirectional imaging device 10. At image processing block 124performs various kinds of image processing on image data. The processor100 comprises the resize block 132. The resize block 132 enlarges orshrinks the size of image data by interpolation. The processor 100comprises a still-image compression block 134. The still-imagecompression block 134 is a codec block for compressing and expanding thestill images such as those in JPEG/TIFF format. The still-imagecompression block 134 is used to store the image data of the generatedspherical image, or to reproduce and output the stored image data. Theprocessor 100 comprises a moving-image compression block 136. Themoving-image compression block 136 is a codec block for compressing andexpanding the moving images such as those in MPEG-4 AVC/H.264 format.The moving-image compression block 136 is used to store the video dataof the generated spherical image, or to reproduce and output the storedvideo data. The processor 100 includes power controller 137.

The image data transferrer 126 transfers the images processed by theimage processing block 124. The SDRAMC 128 controls the SDRAM 138connected to the processor 100 and temporarily storing image data duringimage processing by the processor 100. The memory card control block 140controls data read and write to a memory card and a flash ROM 144inserted to a memory card throttle 142 in which a memory card isdetachably inserted. The USB block 146 controls USB communication withan external device such as personal computer connected via a USBconnector 148. The peripheral block 150 is connected to a power switch166.

The audio unit 152 is connected to a microphone 156 for receiving anaudio signal from a user and a speaker 154 for outputting the audiosignal, to control audio input and output. The serial block 158 controlsserial communication with the external device and is connected to awireless NIC (network interface card) 160. The LCD driver 162 is a drivecircuit for the LCD 164 and converts the image data to signals fordisplaying various kinds of information on an LCD 164. In addition towhat is shown in FIG. 3, video interfaces such as HDMI (High-DefinitionMultimedia Interface) (registered trademark) are may be included and thelike.

The flash ROM 144 stores a control program written in a code that can bedecoded by the CPU 130 and various parameters. When a power supply isturned on by operating the power switch 166, the control program isloaded to a main memory, and the CPU 130 controls operations of therespective units of the device according to the program read into themain memory. Concurrently, the SDRAM 138 and a local SRAM (Static RandomAccess Memory) temporarily store data required for control. By usingrewritable flash ROM 144, the control program and the parameter forcontrol can be changed, and a version of the function can be easilyupdated.

FIG. 4 is a block diagram of a hardware configuration of the informationprocessing device 50. In various implementations, the informationprocessing device 50 may be a mobile device such as a smartphone and atablet computer, a personal computer, a workstation, a server computer,a computer system, and the like. The information processing device 50illustrated in FIG. 4 includes a CPU 52, a random access memory (RAM)54, and hard disk drive (HDD) 56. The CPU 52 controls the operations ofcomponents of the information processing device 50, or controls theoverall operations of the information processing device 50. Informationprocessing device 50 may include circuitry or processing circuitry, suchas CPU 52. The RAM 54 provides the work area of the CPU 52. The HDD 56stores therein an operating system and a control program, such as anapplication, that executes processes in the information processingdevice 50 according to the present implementation, each of the operatingsystem and the control program being written in a code decodable by theCPU 52. The information processing device 50 may include Solid StateDrive (SSD) instead of HDD.

The information processing device 50 may include an input device 58, anexternal storage 60, a display 62, a wireless NIC 64, and a USBconnector 66. The input devices 58 are input devices, such as a mouse, akeyboard, a touchpad, and a touchscreen, and provide a user interface.The external storage 60 is a removable recording medium mounted, forexample, in a memory card slot, and records various types of data, suchas image data in a video format and still image data.

The display 62 performs the display of an operation screen, the displayof the monitor image of the image captured by the omnidirectionalimaging device 10 that is ready to capture or is capturing an image, andthe display of the stored video or still image for reproducing orviewing. The display 62 and the input device 58 enable, through theoperation screen, making instructions for image capturing or changingvarious kinds of setting in the omnidirectional imaging device 10.

The wireless NIC 64 establishes a connection for wireless LANcommunication with an external device such as the omnidirectionalimaging device 10. The USB connector 66 provides a USB connection to anexternal device such as the omnidirectional imaging device 10. By way ofexample, the wireless NIC 64 and the USB connector 66 are described.However, limitation to any specific standard is not intended, andconnection to an external device may be established through anotherwireless connection such as Bluetooth (registered trademark) andwireless USB or through a wired connection such as wired local areanetwork (LAN).

When power is supplied to the information processing device 50 and thepower thereof is turned on, the program is read from a ROM or the HDD56, and loaded into the RAM 54. The CPU 52 follows the program read intothe RAM 54 to control the operations of the parts of the device, andtemporarily stores the data required for the control in the memory. Thisoperation implements functional units and processes of the informationprocessing device 50, as will be described later. As examples of theprogram include an application for giving various instructions to theconnected the omnidirectional imaging device 10 and requesting an imagethrough a bus 68.

<Entire Image Processing>

FIG. 5 illustrates a flow chart of an entire image processing performedby omnidirectional imaging device 10. As illustrated in FIG. 5, thesolid-state image sensors 22A and 22B capture images under a certainexposure condition and output them. Then, the ISPs 108A and 108B performa first image signal processing on the images from the solid-state imagesensors 22A and 22B. As the first image signal processing, any ofoptical black (OB) correcting processing, defective pixel correctionprocessing, Linear correcting processing, shading correcting processing,and area division average processing are performed, and the results ofthe first image signal processing are stored in the memory 300.

The area division average processing is processing for dividing an imagearea included in the captured image into a plurality of areas andcalculating an integration value (or integration average value) ofluminance for each divided area. The results of this processing are usedin the AE control processing.

After the first image signal processing (ISP1) the ISPs 108A and 108Bfurther perform a second image signal processing to the images and theimages are stored in the memory 300. The ISPs 108A and 108B perform anyof white balance 176, Bayer interpolation, color correction, gamma (γ)correction, YUV conversion and edge enhancement (Y CF LT) as the secondimage signal processing.

A color filter of one of colors of red (R), green (G), and a blue (B) isattached on a photodiode on each of the solid-state image sensors 22Aand 22B. The color filter accumulates a light amount from an object.Since the amount of light to be transmitted varies according to thecolor of the filter, an amount of charges accumulated in the photodiodevaries. The color having the highest sensitivity is G, and thesensitivity of R and B is lower than and about half of the sensitivityof G. In the white balance (WB) processing 176, processing for applyinggains to R and B is performed to compensate the differences in thesensitivity and to enhance whiteness of the white in the captured image.Furthermore, since a color of an object changes according to a lightsource color (for example, sunlight and fluorescent light), a functionis provided for changing and controlling the gains of R and B so as toenhance whiteness of the white even when the light source is changed.The parameter of the white balance processing is calculated based onintegration value (or accumulation average value) data of RGB for eachdivided area calculated by the area division average processing.

In the ISP 108A, relative to a Bayer RAW image output from thesolid-state image sensor 22A, the first image signal processing isperformed. The image is stored in the memory 300. In the ISP 108B,similarly, relative to a Bayer RAW image output from the solid-stateimage sensor 22B, the second image signal processing is performed. Theimage is stored in the memory 300.

An automatic exposure control unit 170 performs processing to set theexposure of each of the solid-state image sensor 22A and the solid-stateimage sensor 22B is set to a proper exposure by using an area integratedvalue obtained by the area division average processing so that thebrightness at the image boundary portion of the two images are similarto each other. (it means compound-eye AE). Each of the solid-state imagesensors 22A and 22B may has an independent simple AE processingfunction, and each of the solid-state image sensor 22A and thesolid-state image sensor 22B can independently set a proper exposure. Ina case where change in an exposure condition of each of the sensors Aand B is reduced and the exposure condition is stable, a process shiftsto compound-eye AE control for two images from both solid-state imagesensors. The automatic exposure control unit 170 may be executed on oneISP 108 or the automatic exposure control unit 170 may be distributedmounted on both ISPs 108A and 108B and exchanges information with eachother while considering the information of the other ISP, and theexposure condition parameter of the own solid-state image sensor 22 maybe determined.

As the exposure condition parameters, shutter speed, ISO sensitivity,and aperture value and the like can be used, but the aperture value maybe fixed value. In compound-eye AE, by setting the shutter speeds of thesolid-state image sensor 22A and 22B to be the same, a moving objectacross the solid-state image sensors 22A and 22B can be satisfactorilyconnected. The exposure condition parameters for the solid-state imagesensors 22A and 22B are set from the automatic exposure control unit 170to AE registers 172A and 172B of the solid-state image sensors 22A and22B.

With respect to the solid-state image sensor 22A, of which the firstimage signal processing has been performed, the second image signalprocessing including a white balance processing 176A is performed. Theprocessed data is stored in the memory 300. Similarly, with respect tothe solid-state image sensor 22B, of which the first image signalprocessing has been performed, the second image signal processingincluding a white balance processing 176B is performed. The processeddata is stored in the memory 300. Based on the integration value data ofRGB for each divided area calculated by the area dividing averageprocess, the white balance value calculation unit 174 calculates theparameters of the white balance processing in each of the solid-stateimage sensors 22A and 22B.

The image data after the second image signal processing is sent to thedistortion correction and image synthesis block 118 and the distortioncorrection and image synthesis block 118 performs the distortioncorrection/synthesizing operation and an omnidirectional image isgenerated. Then, based on the information received from the motionsensor 120, the distortion correction/synthesizing operation performsvertical correction representing inclination correction. When the imageis a still image, for example, the image is appropriately JPEGcompressed in the still-image compression block 134 shown in FIG. 3, andthe data is stored in the memory 300, and a file is stored (tagging).When the image is a moving image, for example, the image isappropriately converted into a moving image format such as MPEG-4AVC/H.264 at the moving-image compression block 136 shown in FIG. 3, andthe data is stored in the memory 300, and a file is stored (tagging). Inaddition, the data may be stored in a medium such as an SD card. Thedata is transferred to the information processing device 50 assmartphone (mobile terminal and the like) using wireless LAN (Wi-Fi),Bluetooth (registered trademark), and the like.

Hereinafter, a description relating to generation of an omnidirectionalimage and the generated omnidirectional image is provided with referenceto FIGS. 6A-6C. FIG. 6A is an illustration of data structure of eachimage and the data flow of the image in generating an omnidirectionalimage.

First, images directly captured by each of the solid-state image sensors22A and 22B roughly cover a hemisphere of the whole sphere as a field ofview. Light that passes through each lens system 20A/20B is focused onthe light receiving area of the corresponding solid-state image sensor22A/22B to form an image according to a predetermined projection system.The solid-state image sensor 22A/22B is a two-dimensional image sensordefining a planar area of the light receiving area. Accordingly, theimage formed by the solid-state image sensor 22A/22B is image datarepresented by a plane coordinate system. Such a formed image is atypical fish-eye image that contains an image circle as a whole in whicheach captured area is projected, as illustrated in a partial image A anda partial image B in FIG. 6A.

A plurality of the partial images captured by the plurality ofsolid-state image sensors 22A and 22B is then subjected to distortioncorrection and synthesis processing to form an omnidirectional image(spherical image). In the synthesis processing, an omnidirectionalimage, which constitutes a complementary hemispherical portion, isgenerated from each planar partial image. Then, the images including therespective hemispherical portions are joined together by a stitchingprocessing by matching the overlapping areas of the hemisphericalportions, and the omnidirectional images are synthesized to generate afull omnidirectional image including a whole sphere. The images of therespective hemispherical portions includes overlapping areas, but in thesynthesis process the overlapping areas are blended to look the jointnaturally between the two images

FIG. 6B is an illustration of a plane data structure of the image dataof an omnidirectional image and FIG. 6C is an illustration of aspherical data structure of the image data of an omnidirectional image.

As illustrated FIG. 6B, the image data in the omnidirectional image isrepresented by an array of pixel values having coordinates of a verticalangle φ that corresponds to an angle relative to a predetermined axis,and a horizontal angle θ that corresponds to a rotation angle around theaxis. The horizontal angle θ is represented in the range of 0 to 360degree (or −180 degree to +180 degree), and the vertical angle φ isrepresented in the range of 0 to 180 degree (or −90 degree to +90degree).

As illustrated in FIG. 66C, every pair of coordinates values (θ, φ) onan omnidirectional image format is associated with a point on thespherical surface representing omni-azimuth having the imaging point asthe center, and the omni-azimuth is mapped onto the entire celestialsphere image. The relationship between plane coordinates of an imagecaptured by the fisheye lens, and coordinates on the spherical surfacein the entire celestial sphere image can be associated with each otherusing a predetermined conversion table. The conversion table isgenerated beforehand at a manufacturer or the like, based on design dataor the like of the respective lens optical systems, and following apredetermined projection model, and the data is used for converting apartial image into an omnidirectional image.

In the description of FIG. 6A, the partial image A captured by the lensA is on the center and the partial image B captured by the lens B isseparated and arranged on the left and right. However, as will bedescribed later, the partial image A may be on the left side and thepartial image B may be arranged on the right side.

The following describes the zenith correction and the rotationcorrection using information from the motion sensor 120 with referenceto FIGS. 7A, 7B and 8A-8D.

FIG. 7A shows the definition of the attitude viewed from the side of theomnidirectional imaging device 10, and FIG. 7B shows the definition ofthe attitude viewed from the front of the omnidirectional imaging device10.

As shown in FIGS. 7A and 7B, device angle of the omnidirectional imagingdevice 10 are defined as roll, pitch and yaw. As the optical axisdirection passing through the center of the two lenses of theomnidirectional imaging device 10 is as the front-rear direction, arotation angle (roll) is an angle around an axis and the front-reardirection of the omnidirectional imaging device 10, a rotation angle(pitch) is an angle around an axis about the left-right direction of theomnidirectional imaging device 10 and a rotation angle (yaw) is an anglearound an axis about the top-bottom direction of the omnidirectionalimaging device 10. A rotation that bows the omnidirectional imagingdevice 10 with one lens (for example, the lens opposite to the sidewhere the shutter button 18 is located) as the front surface isrepresented by a pitch. A lateral rotation around the lens optical axisof the omnidirectional imaging device 10 is represented by a roll, and arotation around the central axis of housing of the omnidirectionalimaging device 10 is represented by a yaw.

In the implementation to be described, the front and rear of theomnidirectional imaging device 10 are defined for convenience asfollows. That is, the lens system 20A on the side opposite to theshutter button 18 is a front lens, and the side photographed with thefront lens is the front (F) side. In addition, the lens system 20B onthe side where the shutter button 18 is provided is a rear lens, and theside photographed with the rear lens is a rear (R) side.

FIGS. 8A-8D illustrate the zenith correction and the rotation correctionapplied to an omnidirectional image according to an implementation ofthe present application. In particular, FIG. 8A is a diagram of aspherical data structure before zenith and rotation correction and FIG.8C is an omnidirectional image before the zenith and rotationcorrection. FIG. 8B is the spherical data structure after the zenith androtation correction and FIG. 8D is the omnidirectional image after thezenith and rotation correction.

As described above, the image data of an omnidirectional image format isexpressed as an array of pixel values where the vertical angle φcorresponding to the angle with reference to a certain axis z0 and thehorizontal angle θ corresponding to the angle of rotation around theaxis z0 are the coordinates. If no correction is made, the certain axisz0 is defined with reference to the omnidirectional imaging device 10.For example, the axis z0 is defined as the central axis z0, whichdefines the horizontal angle θ and the vertical angle φ, passing throughthe center of the housing 14 from the bottom to the top where the top isthe imaging body 12 side and the bottom is the opposite side of theomnidirectional imaging device 10 in FIG. 2. Further, for example, thehorizontal angle θ of an omnidirectional image is defined such that thedirection of the optical axis of the optical element of one of the lenssystem 20A and 20B is positioned at the center of the correspondingsolid-state image sensor 22 at the horizontal angle θ.

The zenith correction (correction in the direction of roll and thedirection of pitch) is a correction processing that corrects theomnidirectional images (FIG. 8C) captured with the central axis actuallyinclined with respect to the direction of gravity as illustrated in FIG.8A, to an omnidirectional image (FIG. 8D) captured with the central axisaligned with the direction of gravity as illustrated in FIG. 8B. Therotation correction is a correction (correction in the direction of yaw)that rotates around the direction of gravity in the omnidirectionalimage (FIG. 8D) to which the zenith correction has been made to have thecentral axis aligned with the direction of gravity. The rotationcorrection may not be performed, but is applied, for example, when theomnidirectional image is fixed in a specific direction. Additionally, ina moving image, the rotation correction is preferably used whencorrecting so that the first orientation is always at the center of theomnidirectional image, and when magnetic north is always at the centerof the omnidirectional image.

In an exemplary flow of processing, a partial image is converted into animage including each hemispherical portion, and the obtained images arecombined to generate an omnidirectional image, and the zenith androtation processing (correction) is performed on the generatedomnidirectional image. However, the order of the conversion process, thesynthesis process, and the zenith and rotation process is notparticularly limited.

Additionally, a synthesis process may be performed after zenith androtation correction on each of partial image A and partial image B (andtwo omnidirectional images including complementary hemispherical partsconverted from each partial image). And as a different example, inaddition to rotating coordinate conversion for images in theomnidirectional image format, the conversion table for convertingpartial images to omnidirectional images reflects the effects of zenithand rotation correction, and based on the corrected conversion table,the corrected omnidirectional image can be directly obtained from thepartial image A and the partial image B.

An example of an omnidirectional image displayed on a plane will bedescribed.

FIGS. 9A-9C illustrate a specific example of an omnidirectional imagehaving a difference in brightness according to an implementation of thepresent implementation. Here, there is an equirectangular as one ofprojection methods for displaying an omnidirectional image on a plane.The equirectangular is an image format in which the three-dimensionaldirection of each pixel of a spherical image is decomposed into latitudeand longitude, and pixel values corresponding to a square lattice arearranged. By taking the earth as an example as illustrated in FIGS. 8Cand 8D, when the equirectangular is projected so that the parallels andmeridians intersect at right angles and at equal intervals.

FIG. 9A is an example showing the arrangement of the omnidirectionalimage. The arrangement of the omnidirectional image shows the partialimage A captured by the solid-state image sensor 22A is arranged in thecenter, and the partial image B captured by the solid-state image sensor22B is divided into two and arranged on the left and right of thepartial image A.

FIG. 9B is an example of an omnidirectional image captured by thecompound-eye AE control for the solid-state image sensors 22A and 22B.The partial image B in FIG. 9B is image with low brightness (darkness)overall, which is captured by the solid-state image sensor 22B mainlycapturing in the vehicle. On the other hand, the partial image A in FIG.9B is an image with high brightness (light) overall, which is capturedby the solid-state image sensor 22A mainly capturing outdoor images. Asdescribed above, when the contrast of light and darkness in the image islarge, when the image is taken by the compound-eye AE control, whiteoutor blackout occurs in the image. For example, in FIG. 9B, a part of anoutdoor building is whiteout like an area 500 indicated by a brokenline, resulting in deterioration in image quality.

Here, as a method for improving the image quality when there is a largecontrast in brightness in one image, there is a method of image capture(“shooting”) with HDR processing. HDR processing can improve the imagequality of an image by performing a process of combining a plurality ofimages shot with different exposures when there is a large difference inbrightness in one image.

FIG. 9C shows an omnidirectional image captured by performingcompound-eye AE control on the solid-state image sensors 22A and 22B andperforming HDR processing. A scene in FIG. 9C is same scene as thatshown in FIG. 9B, but the scene in FIG. 9C is shot with HDR processing.Therefore, the building in the area 501 indicated by the broken line inFIG. 9C is captured more clearly than that in FIG. 9B, and the imagequality of the omnidirectional image is improved.

On the other hand, when shooting is performed by combining thecompound-eye AE control and the HDR processing as shown in FIG. 9C, theimage quality may be deteriorated if the subject is moving. Therefore,it is preferable that each of the solid-state image sensor isindependently set to an appropriate exposure (independent AE control).That is, in the independent AE control, an imaging condition of thesolid-state image sensor 22A is determined based on a photometric valueobtained independently by the solid-state image sensor 22A, an imagingcondition of the solid-state image sensor 22B is determined based on aphotometric value obtained independently by the solid-state image sensor22B. In particular, when the difference in brightness of the subjectimaged by each solid-state image sensor is larger than a predeterminedthreshold, shooting is performed according to a shooting mode byindependent AE control (hereinafter referred to as “brightnessdifference scene mode”), which is switched from a normal shooting modeby compound-eye AE control. Note that the “normal shooting mode” hererefers to a shooting mode in which imaging conditions common to eachsolid-state image sensor are set by compound-eye AE control.

FIG. 10 is a diagram for illustrating a functional configuration of theomnidirectional imaging device 10. In an exemplary implementation, theprocessing circuitry of the omnidirectional imaging device 10 may beconfigured to perform the functions of shooting mode switching unit 201,automatic exposure control unit 170 and white balance value calculationunit 174.

The shooting mode switching unit 201 switches the normal shooting modeand the brightness difference scene mode. The shooting mode switchingunit 201 switches the shooting mode based on the brightness betweenpartial images obtained by the solid-state image sensors 22A and 22B.

When the brightness difference scene mode is selected, the automaticexposure control unit 170 sets imaging conditions for appropriateexposure for the solid-state image sensors 22A and 22B based on thephotometric values obtained by the image sensors 22A and 22B. Noted thatthe acquisition of the photometric value is not necessarily performed bythe solid-state image sensor 22, but may be performed by a photometricsensor or the like. In this case, the omnidirectional imaging device 10comprises a photometric sensor corresponding to each solid-state imagesensor 22A and 22B. That is, the omnidirectional imaging device 10comprises a photometric sensor that measures a photometric value forsetting the imaging condition of the solid-state image sensor 22A and aphotometric sensor that measures a photometric value for setting theimaging condition of the solid-state image sensor 22B. The imagingconditions to be set can include various parameters such as shutterspeed, ISO sensitivity, aperture value and the like.

Each shooting mode can be selected from manually operating applicationsof the omnidirectional imaging device 10 or the information processingdevice 50 by input of a user, but the implementation is not particularlylimited those. For example, the shooting mode switching unit 201 mayautomatically switch to set the brightness difference scene when theomnidirectional imaging device 10 or the information processing device50 compares the photometric values acquired by the solid-state imagesensor 22 or the photometric sensor, and the difference is larger than apredetermined threshold value.

Similarly, in the brightness difference scene mode, a white balancevalue calculation unit 174 calculates the white balance value of eachsolid-state image sensor 22A and 22B based on the information acquiredby each of the solid-state image sensors 22A and 22B at the shooting,and the white balance value calculation unit 174 performs white balanceprocessing.

In addition, when switching from the normal shooting mode to thebrightness difference scene mode, initial values of the imagingcondition of each solid-state image sensor in brightness differencescene mode may be set to the image condition set in the normal shootingmode at the time of switching. Thereby, a convergence of the imagingcondition by feedback control is accelerated.

Next, processing for selecting a shooting mode will be described withrespect to the process performed by processing circuitry of theomnidirectional imaging device 10 and illustrated in FIG. 11. Theomnidirectional imaging device 10 starts processing from step S101. Instep S102, the shooting mode switching unit 201 acquires information ona brightness of an image captured by each of solid-state image sensors22A and 22B.

In step S103, the shooting mode switching unit 201 calculates abrightness difference between the two images based on the informationacquired in step S102. When the brightness difference is larger than thethreshold (YES in S103), the shooting mode switching unit 201 determinesto proceed to step S104, and the shooting mode switching unit 201 thenselects the brightness difference scene mode for performing imagecapture by the omnidirectional imaging device 10. When the brightnessdifference is smaller than the threshold (NO in S103), the shooting modeswitching unit 201 determines to proceed to step S105, and the shootingmode switching unit 201 selects the normal shooting mode for performingimage capture by the omnidirectional imaging device 10. Thereafter, theprocess ends in step S106.

Next, an exemplary omnidirectional image captured in the brightnessdifference scene mode will be described. FIGS. 12A-12C is a diagramillustrating an example of an omnidirectional image captured in thebrightness difference scene mode. When shot in the brightness differencescene mode, an omnidirectional image in which partial images arearranged side by side in one direction (hereinafter referred to as“horizontal parallel arrangement image”) is preferably output as shownin FIG. 12A. FIG. 12A shows an example of an omnidirectional image inwhich the partial image A captured by the solid-state image sensor 22Ais arranged on the left side and the partial image B captured by thesolid-state image sensor 22B is arranged on the right side.

With such an arrangement, the entire image can be fit well. The numberof boundary portions with a large difference in brightness can bereduced, an uncomfortable feeling that the user sees the image isreduced. Further, for a horizontally long omnidirectional image, byarranging the partial images side by side, the length of the boundarybetween the partial images can be set to the length of the short sidedirection of the omnidirectional image. That is, since the length of theboundary can be shortened, the uncomfortable feeling that the user seesthe image is reduced.

FIG. 12B shows a specific example of the omnidirectional image capturedin the brightness difference scene mode. By setting the imagingconditions of each image sensors 22 by independent AE control andshooting, overexposure and underexposure due to contrast are reduced.For example, even if one image sensor mainly shoots a person inside thevehicle, and the other image sensor mainly shoots landscapes of theoutside of the vehicle, as shown in FIG. 12B.

In addition, in the case of an omnidirectional image captured in thebrightness difference scene mode, the processor 100 may perform controlto invalidate the connection position detection processing function ofthe distortion correction and image synthesis block 118, and may notperform the connection position detection processing for the overlappingareas of the partial images. Since the two partial images shot in thebrightness difference scene mode have a large light/dark difference,joining them after the performing connection position detectionprocessing makes the connected portions unnatural and causes adeterioration in image quality. Therefore, the omnidirectional imagethat looks more natural is generated by not performing the connectionposition detection process of partial images.

Further, when the contrast difference is particularly remarkable in thebrightness difference scene mode, the processing circuitry of anomnidirectional imaging device 10 may perform a process to insert aboundary line at a boundary portion of a partial image at the time ofoutputting the horizontal parallel arrangement image as shown in FIG.12C. For example, in FIG. 12C, the difference in the vicinity of theboundary is made inconspicuous by outputting the horizontal parallelarrangement image in which the boundary line is inserted. Thus, when thedifference between partial images is remarkable, it can be made torecognize that the separate image is displayed side by side with respectfor user. In addition, other images other than the boundary line may beinserted in the boundary portion of the partial image.

Further, when an omnidirectional image is output in the brightnessdifference scene mode, depending on the rotation direction of thedevice, the visibility may be impaired by performing a zenithcorrection. In particular, an orientation of the omnidirectional imagingdevice 10 may be considered when performing a zenith correction in orderto yield a correct orientation of output images for better visibility ofa user. For example, there may be a situation in which it is preferablenot to perform the zenith correction for rotation in the pitchdirection, but to perform zenith correction for rotation in the rolldirection. FIGS. 13A-13C illustrate orientations of omnidirectionalimaging device with respect to zenith correction of orientations ofobjects in captured omnidirectional images.

The left figure of FIG. 13A is a figure which shows a first example ofcapturing by the omnidirectional imaging device 10. An example ofimaging is shown that the image sensor 22A shoot the car of the frontside, and the image sensor 22B shoots the person of the rear side.

In the left diagram of FIG. 13A, the omnidirectional imaging device 10does not rotate in any of the yaw direction, the roll direction, and thepitch direction, and performs imaging in an upright posture.

When imaged in the upright posture, the generated horizontal parallelarrangement image is an omnidirectional image as shown in the rightfigure of FIG. 13A. That is, the left partial image A of theomnidirectional image includes a car image, and the right partial imageB includes a person image. In such a case, since the omnidirectionalimaging device 10 is imaging in the upright posture, the zenithcorrection need not be performed.

The case is that the omnidirectional imaging device 10 is rotated in theroll direction and imaged. As shown in the left figure of FIG. 13B, whenthe omnidirectional imaging device 10 is rotated 90 degrees about thelens optical axis in the roll direction, each partial image of thehorizontal parallel arrangement image is shown in the center of FIG.13B.

FIG. 13B shows a subject (such as a person or a car) in the partialimage rotates. When zenith correction is performed in the case, thehorizontal parallel arrangement image as shown in the right figure ofFIG. 13B can be generated to be similar to the right diagram of FIG.13A. In such a case, since the subject such as a car or a person islocated near the optical axis, the amount of distortion in the partialimage is small. Therefore, even when distortion correction is performedalong with zenith correction, the distortion of the subject is small andimage quality is unlikely to deteriorate.

The case is that the omnidirectional imaging device 10 is rotated in thepitch direction and imaged. As shown in the left figure of FIG. 13C,when the omnidirectional imaging device 10 is rotated in the pitchdirection and imaged so that the lens optical axis is directed in thevertical direction, the partial image A of the horizontal parallelarrangement image includes the upper part of the car and the person, andthe partial image B includes the lower part of the car and the person asshown in the center of FIG. 13C.

When zenith correction is performed in the case, the omnidirectionalimage includes a car on the left side and a person on the right side asshown in the right figure of FIG. 13C. However, the zenith-correctedimage as shown in FIG. 13C is not a horizontal parallel arrangementimage, and the boundary of the partial image occurs in the directionalong the long side of the image. Therefore, an unnatural image with along boundary is obtained. In addition, if distortion correction isperformed with zenith correction, subjects such as cars and peopleappear in positions away from the lens optical axis (near the edge ofthe partial image), so the amount of distortion increases and imagequality decreases.

Therefore, it is preferable that the omnidirectional imaging system 1 ofthe present implementation performs a zenith correction for rotation inthe roll direction and does not perform a zenith correction for rotationin the pitch direction. Note that rotation correction may or may not beperformed for rotation in the yaw direction.

In the implementations described, the description has been made mainlyusing still images as examples, but the implementations are notparticularly limited. Therefore, the implementation described for notonly still images but also moving images may be applied.

As described above, according to the implementations, it is possible toprovide an imaging device, an imaging system, a method, and anon-transitory computer readable medium storing executable instructionsor a program for reducing a sense of discomfort of the omnidirectionalimage and improve the image quality regardless of the subject or thescene.

In the implementation described above, the omnidirectional imagingsystem 1 is described using the omnidirectional imaging device 10 andthe omnidirectional imaging device 10 including the informationprocessing device 50 communicating with the omnidirectional imagingdevice 10 as examples. However, the configuration of the omnidirectionalimaging system 1 is not limited to the configuration described above.Therefore, not all the functional means described in the implementationsare necessarily included in the omnidirectional imaging device 10. Forexample, in other implementations, the system of the aboveimplementation may be realized by the cooperation between theomnidirectional imaging device 10 and the information processing device50. In addition, the system of the above implementation may be an imageprocessing system that processes an image captured by an externalimaging unit.

The functions of the omnidirectional imaging system can be achieved by acomputer-executable program written in legacy programming language suchas assembler, C, C++, C #, JAVA(registered trademark) or object-orientedprogramming language. Such a program can be stored in a storage mediumsuch as ROM, EEPROM, EPROM, flash memory, flexible disc, CD-ROM, CD-RW,DVD-ROM, DVD-RAM, DVD-RW, blue ray disc, SD card, or MO and distributedthrough an electric communication line. Further, a part or all of theabove functions can be implemented on, for example, a programmabledevice (PD) as field programmable gate array (FPGA) or implemented asapplication specific integrated circuit (ASIC). To realize the functionson the PD, circuit configuration data as bit stream data and datawritten in HDL (hardware description language), VHDL (very high speedintegrated circuits hardware description language), and Verilog-HDLstored in a storage medium can be distributed.

A non-transitory recording medium storing a computer-readable code forcontrolling a computer system to carry out an image processing methodincludes: acquiring a plurality of images and determining whether adifference in brightness between the plurality of images exceeds apredetermined threshold; in a case that the determining determines thatthe difference in brightness does not exceed the predeterminedthreshold, combining the plurality of images to generate an outputimage; and in a case that the determining determines that the differencein brightness exceeds the predetermined threshold, the method furtherincludes arranging the plurality of images side by side in one directionto generate an output image.

Although the present application has been described in terms ofexemplary implementations, it is not limited thereto. It should beappreciated that variations or modifications may be made in theimplementations described by persons skilled in the art withoutdeparting from the scope of the present application as defined by thefollowing claims.

Numerous additional modifications and variations are possible in lightof the above teachings. It is therefore to be understood that, withinthe scope of the above teachings, the present disclosure may bepracticed otherwise than as specifically described herein. With someembodiments having thus been described, it will be obvious that the samemay be varied in many ways. Such variations are not to be regarded as adeparture from the scope of the present disclosure and appended claims,and all such modifications are intended to be included within the scopeof the present disclosure and appended claims.

What is claimed is:
 1. An imaging device comprising: a plurality ofimage sensors configured to capture a plurality of images, andprocessing circuitry configured to: set an exposure condition when abrightness difference scene mode is selected by switching from a normalshooting mode, when a difference in brightness between the plurality ofimages exceeds a predetermined threshold; determine whether thedifference in brightness between the plurality of images exceeds apredetermined threshold using the exposure condition setting; and basedon the determination that the difference in brightness exceeds thepredetermined threshold, arrange the plurality of images side by side inone direction to be output as an output image.
 2. The imaging deviceaccording to claim 1, wherein based on the determination that thedifference in brightness exceeds the predetermined threshold, theprocessing circuitry is configured to perform zenith correction, aroundan optical axis of each of the image sensors, to correct a correspondingone of the plurality of images.
 3. The imaging device according to claim1, wherein based on the determination that the difference in brightnessexceeds the predetermined threshold, the processing circuitry isconfigured to generate an output image having a boundary line betweenthe plurality of images captured by the plurality of image sensors. 4.The imaging device according to claim 1, wherein each of the pluralityof image sensors is configured to capture a wide-angle lens image or afish-eye lens image, and wherein the processing circuitry is configuredto generate the output image that is an omnidirectional image based onthe wide-angle lens image or the fish-eye lens image.
 5. The imagingdevice according to claim 1, wherein the plurality of image sensorsincludes a plurality of imaging optical systems, and wherein imagingsurfaces of the plurality of imaging optical systems are disposed facingin opposite directions and optical axes of the imaging optical systemsare matched.
 6. An image processing system comprising: processingcircuitry configured to: acquire a plurality of images captured by aplurality of image sensors; set an exposure condition when a brightnessdifference scene mode is selected by switching from a normal shootingmode, when a difference in brightness between the plurality of imagesexceeds a predetermined threshold; determine whether the difference inbrightness between the plurality of images exceeds a predeterminedthreshold using the exposure condition setting; and based on thedetermination that the difference in brightness exceeds thepredetermined threshold, arrange the plurality of images side by side inone direction to generate an output image.
 7. An image processing methodcomprising: acquiring a plurality of images, setting an exposurecondition when a brightness difference scene mode is selected byswitching from a normal shooting mode, when a difference in brightnessbetween the plurality of images exceeds a predetermined threshold, anddetermining whether the difference in brightness between the pluralityof images exceeds a predetermined threshold using the exposure conditionsetting; wherein, in a case that the determining determines that thedifference in brightness exceeds the predetermined threshold, the methodfurther comprising arranging the plurality of images side by side in onedirection to generate an output image.
 8. The image processing methodaccording to claim 7, further comprising: capturing the plurality ofimages with a plurality of image sensors.
 9. The image processing systemaccording to claim 6, wherein based on the determination that thedifference in brightness exceeds the predetermined threshold, theprocessing circuitry is configured to perform zenith correction, aroundan optical axis of each of the image sensors, to correct a correspondingone of the plurality of images.
 10. The image processing systemaccording to claim 6, wherein based on the determination that thedifference in brightness exceeds the predetermined threshold, theprocessing circuitry is configured to generate an output image having aboundary line between the plurality of images captured by the pluralityof image sensors.
 11. The image processing system according to claim 6,wherein each of the plurality of image sensors is configured to capturea wide-angle lens image or a fish-eye lens image, and wherein theprocessing circuitry is configured to generate the output image that isan omnidirectional image based on the wide-angle lens image or thefish-eye lens image.
 12. The image processing system according to claim6, wherein the plurality of image sensors includes a plurality ofimaging optical systems, and wherein imaging surfaces of the pluralityof imaging optical systems are disposed facing in opposite directionsand optical axes of the imaging optical systems are matched.
 13. Theimage processing method according to claim 7, wherein based on thedetermining that the difference in brightness exceeds the predeterminedthreshold, performing zenith correction, around an optical axis of eachof the image sensors, to correct a corresponding one of the plurality ofimages.
 14. The image processing method according to claim 7, whereinbased on the determining that the difference in brightness exceeds thepredetermined threshold, generating an output image having a boundaryline between the plurality of images captured by the plurality of imagesensors.
 15. The image processing method according to claim 7, whereinthe acquiring the plurality of images acquires a wide-angle lens imageor a fish-eye lens image, and wherein the generating the output imageperforms generating the output image that is an omnidirectional imagebased on the wide-angle lens image or the fish-eye lens image.
 16. Theimage processing method according to claim 7, wherein the acquiring theplurality of images acquires the plurality of image sensors using aplurality of imaging optical systems, and wherein imaging surfaces ofthe plurality of imaging optical systems are disposed facing in oppositedirections and optical axes of the imaging optical systems are matched.